Methods and systems for determining a state of an unmanned aerial vehicle

ABSTRACT

A method for determining an external state of an unmanned aerial vehicle (UAV) includes obtaining historical external state information of the UAV, obtaining a current image and a historical image captured before the current image, predicting a matching feature point in the current image that corresponds to a target feature point in the historical image according the historical external state information, and determining the external state of the UAV based on the matching feature point.

CROSS-REFERENCE

This application is a continuation application of U.S. application Ser.No. 14/949,794, filed on Nov. 23, 2015, which is a continuation-in-partapplication of International Application No. PCT/CN2014/073985, filed onMar. 24, 2014, the entire contents of both of which are incorporatedherein by reference.

BACKGROUND OF THE DISCLOSURE

Aerial vehicles, such as unmanned aerial vehicles (UAVs), have beendeveloped for a wide range of applications including surveillance,search and rescue operations, exploration, and other fields. Such UAVscan often be controlled and navigated by a remote controller. In someinstances, UAVs can be vehicles capable of sensing and navigatingthrough an environment without guidance from a human controller.

One commonly used autonomous navigating technique is visual navigationby virtue of cameras. The UAVs may autonomously determine its stateinformation, including a height, a speed, a direction, a coordinate anda distance to certain object, during visually navigated flight. However,a deviation may occur between the autonomously determined stateinformation and the actual state. The deviation becomes larger overtime, thus a reliability of the visual navigation may be reduced.

SUMMARY OF THE DISCLOSURE

A need exists for improved methods and systems for determining a stateof UAVs during visual navigation. The present disclosure providesmethods and systems related to determine a state of the UAV by updatingthe determined state information of UAV with a relative proportionalrelationship between a first reference frame (e.g., a local orcamera-based reference frame) or coordinates under a first coordinatesystem (e.g., camera coordinates) and a second reference frame (e.g., aglobal reference frame) or coordinates under a second coordinate system(e.g., world coordinates). The UAV may be provided with a monocularcamera unit, a proximity sensor and one or more processors. The one ormore processors may be configured to determine external stateinformation of the UAV based on image data captured by the monocularcamera unit, and calculate a relative proportional relationship to beapplied to the determined external state information. The relativeproportional relationship may be calculated based at least in part on(1) a predetermined positional relationship between the monocular cameraunit and the proximity sensor and (2) a proximity data about theenvironment of the UAV. The processor may be configured to update thedetermined external state information of the UAV at least by applyingthe relative proportional relationship.

In one aspect of the present disclosure, a method determining anexternal state of an unmanned aerial vehicle (UAV) is described. Themethod may include determining, individually or collectively by one ormore processors onboard the UAV, external state information of the UAVbased on image data of an environment of the UAV captured by a monocularcamera unit onboard the UAV; calculating, individually or collectivelyby the one or more processors onboard the UAV, a relative proportionalrelationship to be applied to the external state information of the UAVbased at least in part on (1) a predetermined positional relationshipbetween the monocular camera unit and a proximity sensor unit and (2)proximity data about the environment of the UAV that is acquired by theproximity sensor unit; and updating, individually or collectively by theone or more processors onboard the UAV, the external state informationof the UAV at least by applying the relative proportional relationshipto the external state.

In some embodiments, the relative proportional relationship may comprisea scale factor.

In some embodiments, the proximity sensor unit may comprise at least oneof a lidar sensor, an ultrasound sensor, or an infrared sensor.

In some embodiments, the proximity data may include a distance to anexternal object within the environment.

In some embodiments, the external state information may comprisedistance information of the UAV relative to one or more external objectswithin the environment. In other embodiments, the external stateinformation comprises position information of one or more externalobjects within the environment or position information of the UAV.

In some embodiments, the external state information may include aplurality of feature points, each associated with a first set ofcoordinates under a first reference frame. Updating the external stateinformation may include applying the relative proportional relationshipto the first set of coordinates associated with at least one of theplurality of feature points to obtain a second set of coordinates undera second reference frame. In some embodiments, the first reference framemay be a local reference frame, and the second reference frame may be aglobal reference frame.

In some embodiments, the proximity data may include proximityinformation for a first feature point of the plurality of feature pointsand may not include proximity information for a second feature point ofthe plurality of feature points. Determining external state informationof the UAV may comprise: extracting the plurality of feature points fromthe image data; and determining the first sets of coordinatesrespectively associated with the plurality of feature points under thefirst reference frame.

In some embodiments, calculating the relative proportional relationshipmay comprise: determining a second set of coordinates under a secondreference frame for the first feature point based on the proximityinformation for the first feature point and the predetermined positionalrelationship between the monocular camera unit and the proximity sensorunit; and using the first set of coordinates and the second set ofcoordinates for the first feature point to determine the relativeproportional relationship.

In some embodiments, updating the external state information maycomprise applying the relative proportional relationship to the firstset of coordinates associated with the second feature point to obtain asecond set of coordinates for the second feature point under a secondreference frame. In some embodiments, the first reference frame may be alocal reference frame, and the second reference frame may be a globalreference frame.

In some embodiments, determining external state information of the UAVmay comprise: obtaining historical state information of the UAV or themonocular camera unit; obtaining the image data comprising a currentimage and an historical image preceding the current image; predicting amatching feature point in the current image corresponding to a targetfeature point in the historical image according to the historical stateinformation; and calculating, based on the matching feature point,external state information of the UAV or the monocular camera unit.Predicting the matching feature point in the current image may comprise:predicting a coordinate position of a target feature point of thehistorical image in the current image according to the historical stateinformation; and selecting a feature point that is closest to thecoordinate position as the matching feature point in the current image.Updating the external state information of the UAV may comprise applyinga Kalman filter to the external state information according to apre-established state model and a pre-established measurement model.

In some embodiments, the monocular camera unit may comprise a monocularpanoramic camera and a reflection mirror for reflecting external lightrays to the monocular camera. The method may further comprisecalibrating the image data according to a pre-determined calibrationmodel before determining the external state information. Calibrating theimage data may comprise obtaining a coordinate mapping relationshipbetween arbitrary point in an external space and a corresponding pointin an image acquired by the monocular camera unit according to thepre-established calibration model corresponding to a reflection mirrorof the monocular camera unit; and modifying the image according to thecoordinate mapping relationship.

In another aspect of the present disclosure, system for determining anexternal state of an unmanned aerial vehicle (UAV) is described. Thesystem may include a monocular camera unit configured to capture imagedata of an environment of the UAV; a proximity sensor unit configured toacquire proximity data about the environment of the UAV; and one or moreprocessors configured to, individually or collectively: determineexternal state information of the UAV based on the image data capturedby the monocular camera unit; calculate a relative proportionalrelationship to be applied to the external state information of the UAVbased at least in part on (1) a predetermined positional relationshipbetween the monocular camera unit and the proximity sensor unit and (2)the proximity data; and update the external state information of the UAVat least by applying the relative proportional relationship to theexternal state information.

In some embodiments, the relative proportional relationship may comprisea scale factor.

In some embodiments, the proximity sensor unit may comprise at least oneof a lidar sensor, an ultrasound sensor, or an infrared sensor.

In some embodiments, the proximity data may include a distance to anexternal object within the environment.

In some embodiments, the external state information may comprisedistance information of the UAV relative to one or more external objectswithin the environment. In other embodiments, the external stateinformation may comprise position information of one or more externalobjects within the environment or position information of the UAV.

In some embodiments, external state information may include a pluralityof feature points, each associated with a first set of coordinates undera first reference frame. Updating the external state information mayinclude applying the relative proportional relationship to the first setof coordinates associated with at least one of the plurality of featurepoints to obtain a second set of coordinates under a second referenceframe. In some embodiments, the first reference frame may be a localreference frame, and the second reference frame may be a globalreference frame.

In some embodiments, the proximity data may include proximityinformation for a first feature point of the plurality of feature pointsand may not include proximity information for a second feature point ofthe plurality of feature points. Determining external state informationof the UAV may comprise: extracting the plurality of feature points fromthe image data; and determining the first sets of coordinatesrespectively associated with the plurality of feature points under thefirst reference frame.

In some embodiments, calculating the relative proportional relationshipmay comprise: determining a second set of coordinates under a secondreference frame for the first feature point based on the proximityinformation for the first feature point and the predetermined positionalrelationship between the monocular camera unit and the proximity sensorunit; and using the first set of coordinates and the second set ofcoordinates for the first feature point to determine the relativeproportional relationship.

In some embodiments, updating the external state information maycomprise applying the relative proportional relationship to the firstset of coordinates associated with the second feature point to obtain asecond set of coordinates for the second feature point under a secondreference frame. In some embodiments, the first reference frame may be alocal reference frame, and the second reference frame may be a globalreference frame.

In some embodiments, determining external state information of the UAVmay comprise: obtaining historical state information of the UAV or themonocular camera unit; obtaining the image data comprising a currentimage and an historical image preceding the current image; predicting amatching feature point in the current image corresponding to a targetfeature point in the historical image according to the historical stateinformation; and calculating, based on the matching feature point,external state information of the UAV or the monocular camera unit.Predicting the matching feature point in the current image may comprise:predicting a coordinate position of a target feature point of thehistorical image in the current image according to the historical stateinformation; and selecting a feature point that is closest to thecoordinate position as the matching feature point in the current image.Updating the external state information of the UAV may comprise applyinga Kalman filter to the external state information according to apre-established state model and a pre-established measurement model.

In some embodiments, the monocular camera unit may comprise a monocularpanoramic camera and a reflection mirror for reflecting external lightrays to the monocular camera. The one or more processors may be furtherconfigured to, individually or collectively, calibrating the image dataaccording to a pre-determined calibration model before determining theexternal state information. Calibrating the image data may compriseobtaining a coordinate mapping relationship between arbitrary point inan external space and a corresponding point in an image acquired by themonocular camera unit according to the pre-established calibration modelcorresponding to a reflection mirror of the monocular camera unit; andmodifying the image according to the coordinate mapping relationship

In another aspect of the present disclosure, one or more non-transitorycomputer-readable storage media having stored thereon executableinstructions is described. The executable instructions that, whenexecuted by one or more processors of a computer system onboard anunmanned aerial vehicle (UAV), may cause the computer system to atleast: determine external state information of the UAV based on imagedata of an environment of the UAV captured by a monocular camera unitonboard the UAV; calculate a relative proportional relationship to beapplied to the external state information of the UAV based at least inpart on (1) a predetermined positional relationship between themonocular camera unit and a proximity sensor unit and (2) proximity dataabout the environment of the UAV that is acquired by the proximitysensor unit; and update the external state information of the UAV atleast by applying the relative proportional relationship to the externalstate information.

In some embodiments, the relative proportional relationship may comprisea scale factor.

In some embodiments, the proximity sensor unit may comprise at least oneof a lidar sensor, an ultrasound sensor, or an infrared sensor.

In some embodiments, the proximity data may include a distance to anexternal object within the environment.

In some embodiments, the external state information may comprisedistance information of the UAV relative to one or more external objectswithin the environment. In other embodiments, the external stateinformation may comprise position information of one or more externalobjects within the environment or position information of the UAV.

In some embodiments, the external state information may include aplurality of feature points, each associated with a first set ofcoordinates under a first reference frame. Updating the external stateinformation may include applying the relative proportional relationshipto the first set of coordinates associated with at least one of theplurality of feature points to obtain a second set of coordinates undera second reference frame. In some embodiments, the first reference framemay be a local reference frame, and the second reference frame may be aglobal reference frame.

In some embodiments, the proximity data may include proximityinformation for a first feature point of the plurality of feature pointsand may not include proximity information for a second feature point ofthe plurality of feature points. Determining external state informationof the UAV may comprise: extracting the plurality of feature points fromthe image data; and determining the first sets of coordinatesrespectively associated with the plurality of feature points under thefirst reference frame.

In some embodiments, calculating the relative proportional relationshipmay comprise: determining a second set of coordinates under a secondreference frame for the first feature point based on the proximityinformation for the first feature point and the predetermined positionalrelationship between the monocular camera unit and the proximity sensorunit; and using the first set of coordinates and the second set ofcoordinates for the first feature point to determine the relativeproportional relationship.

In some embodiments, updating the external state information maycomprise applying the relative proportional relationship to the firstset of coordinates associated with the second feature point to obtain asecond set of coordinates for the second feature point under a secondreference frame. In some embodiments, the first reference frame may be alocal reference frame, and the second reference frame may be a globalreference frame.

In some embodiments, determining external state information of the UAVmay comprise: obtaining historical state information of the UAV or themonocular camera unit; obtaining the image data comprising a currentimage and an historical image preceding the current image; predicting amatching feature point in the current image corresponding to a targetfeature point in the historical image according to the historical stateinformation; and calculating, based on the matching feature point,external state information of the UAV or the monocular camera unit.Predicting the matching feature point in the current image may comprise:predicting a coordinate position of a target feature point of thehistorical image in the current image according to the historical stateinformation; and selecting a feature point that is closest to thecoordinate position as the matching feature point in the current image.Updating the external state information of the UAV may comprise applyinga Kalman filter to the external state information according to apre-established state model and a pre-established measurement model.

In some embodiments, the monocular camera unit may comprise a monocularpanoramic camera and a reflection mirror for reflecting external lightrays to the monocular camera. The method may further comprisecalibrating the image data according to a pre-determined calibrationmodel before determining the external state information. Calibrating theimage data may comprise obtaining a coordinate mapping relationshipbetween arbitrary point in an external space and a corresponding pointin an image acquired by the monocular camera unit according to thepre-established calibration model corresponding to a reflection mirrorof the monocular camera unit; and modifying the image according to thecoordinate mapping relationship

In some embodiments, the executable instructions may further cause thecomputer system to calibrating the image data according to apre-determined calibration model before determining the external stateinformation. Calibrating the image data may comprise obtaining acoordinate mapping relationship between arbitrary point in an externalspace and a corresponding point in an image acquired by the monocularcamera unit according to the pre-established calibration modelcorresponding to a reflection mirror of the monocular camera unit; andmodifying the image according to the coordinate mapping relationship.

It shall be understood that different aspects of the disclosure can beappreciated individually, collectively, or in combination with eachother. Various aspects of the disclosure described herein may be appliedto any of the particular applications set forth below or for any othertypes of movable objects. Any description herein of aerial vehicles,such as unmanned aerial vehicles, may apply to and be used for anymovable object, such as any vehicle. Additionally, the systems, devices,and methods disclosed herein in the context of aerial motion (e.g.,flight) may also be applied in the context of other types of motion,such as movement on the ground or on water, underwater motion, or motionin space.

Other objects and features of the present disclosure will becomeapparent by a review of the specification, claims, and appended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the disclosure are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present disclosure will be obtained by reference tothe following detailed description that sets forth illustrativeembodiments, in which the principles of the disclosure are utilized, andthe accompanying drawings of which:

FIG. 1 shows an example of an unmanned aerial vehicle (UAV) that isvisually navigated by virtue of a monocular camera, in accordance withsome embodiments.

FIG. 2 illustrates a process for determining and updating a state of UAVin accordance with some embodiments.

FIG. 3 illustrates an exemplary calculation of a relative proportionalrelationship which is to be applied on the determined external stateinformation of the UAV, in accordance with an embodiment of thedisclosure.

FIG. 4 illustrates a process for determining an external state of UAVbased on image data, in accordance with an embodiment of the disclosure.

FIG. 5 is a schematic illustration by way of block diagram of a systemfor determining an external state of UAV in accordance with anembodiment of the present disclosure.

FIG. 6 is a flow chart illustrating a method of determining and updatinga state of UAV by applying a Kalman filter, in accordance with anembodiment of the disclosure.

FIG. 7 is a schematic illustration by way of block diagram of a systemfor determining and updating an external state of UAV in accordance withan embodiment of the present disclosure.

FIG. 8 shows an original image captured by the monocular camera, whichis a panoramic type, as an example of image calibration in accordancewith an embodiment of the present disclosure.

FIG. 9 shows a calibrated image after column-expansion calibration as anexample of image calibration in accordance with an embodiment of thepresent disclosure.

FIG. 10 illustrates an exemplary movable object in accordance withembodiments of the present disclosure.

FIG. 11 illustrates a movable object including a carrier and a payload,in accordance with embodiments of the present disclosure.

FIG. 12 is a schematic illustration by way of block diagram of a systemfor controlling a movable object, in accordance with embodiments of thepresent disclosure.

DETAILED DESCRIPTION OF THE DISCLOSURE

The methods and systems described herein provide an effective approachto autonomously determine and update external state information of a UAVduring visual navigation. The visually determined external stateinformation of the UAV, based on image data of an environment of the UAVcaptured by a monocular camera unit onboard the UAV, may be updated byusing a relative proportional relationship. The relative proportionalrelationship may be calculated based on (1) a predetermined positionalrelationship between the monocular camera unit and a proximity sensoronboard the UAV and (2) proximity data about the environment of the UAVthat is acquired by the proximity sensor. In some embodiments, therelative proportional relationship can include a scale factor or atransformation matrix between a first reference frame (e.g., local orcamera-based reference frame) or coordinates under a first coordinatesystem (e.g., camera coordinates) and a second reference frame (e.g.,global reference frame) or coordinates under a second coordinate system(e.g., world coordinates). The updated external state information of theUAV may provide a better estimation of the actual state information ofthe UAV compared with the visually determined external stateinformation, thus enabling more accurate control and navigation for theautonomous flight. With the methods and systems of present disclosure,the deviation between the visually determined external state informationand the actual state of the UAV may be reduced, and the reliability ofthe visual navigation may be improved.

FIG. 1 shows an example of an unmanned aerial vehicle (UAV) 100 that isvisually navigated by virtue of a monocular camera unit in accordancewith an embodiment of the disclosure. The UAV 110 may have a body. Insome instances, the body may be a central body which may have one ormore branching members, or “arms.” The arms may extend outward from thebody in a radial manner and be joined via the body. The number of armsmay match the number of propulsion units, or rotors, of the UAV. Thebody may comprise a housing. The housing may enclose one or morecomponents of the UAV within the housing. In some instances, one or moreelectrical components of the UAV may be provided within the housing. Forexample, a flight controller of the UAV may be provided within thehousing. The flight controller may control operation of one or morepropulsion units of the UAV.

The monocular camera unit 120 may be rigidly coupled to the UAV 110.Alternatively, the monocular camera unit 120 may be permitted to moverelative to the UAV 110 with respect to up to six degrees of freedom.The monocular camera unit 120 may be directly mounted onto the UAV 110,or coupled to a support structure mounted onto the UAV 110. In someembodiments, the monocular camera unit 120 may be an element of apayload of the UAV 110. Since the monocular camera unit 120 is coupledto the UAV 110, the external state information of the monocular cameraunit 120 may be considered as the external state of the UAV 110.

The monocular camera unit 120 may capture images of an environment ofthe UAV 110. The monocular camera unit 120 may capture images at aspecified frequency to produce a series of image data over time. Theseries of image data obtained from the monocular camera unit 120 overtime may be processed by a processor or processors to determine theexternal state information (e.g., position, orientation, and/orvelocity) of the UAV 110 using any suitable method, such as a machinevision algorithm. For example, a machine vision algorithm can be used toidentify one or more feature points within each image (e.g., an edge ofan object, a corner of an object, or a boundary between objects of twodifferent colors). Any suitable method or combination of methods may beused to identify and provide a digital representation of the featurepoints, such as the features from accelerated segment test (FAST)algorithm or the binary robust independent elementary features (BRIEF)algorithm. The image data may then be matched to each other to identifya set of common feature points appearing in the series of image data.The external state information of the UAV 110 may be determined based onthe common feature points and a time interval between two images.

The UAV 110 may carry a proximity sensor 130 onboard to obtain aproximity data about the environment of the UAV 110. In some instances,the proximity sensor 130 may be used to measure a distance between theproximity sensor 130 and a target external to the UAV 110. Since theproximity sensor 130 is onboard the UAV 110, the distance between theproximity sensor 130 and a target external may be considered as thedistance between the UAV 110 and the target external. The proximitysensor 130 may be a lidar sensor, an ultrasonic sensor, or an infraredsensor. The proximity sensor 130 may be directly mounted onto the UAV110, or coupled to a support structure mounted onto the UAV 110. In someembodiments, the proximity sensor 130 may be an element of a payload ofthe UAV 110.

In some embodiments, the proximity sensor 130 may be rotated (e.g.,rotated 360°) to obtain proximity data for a plurality of objectsexternal to the UAV 110. The proximity data for the external objects maybe analyzed to determine at least in part a relative proportionalrelationship. The relative proportional relationship may be used toupdate the determined external state information of the UAV 110, whichis determined based on image data captured by the monocular camera, toobtain updated external state information of the UAV 110 for precisecontrol and navigation during the autonomous flight.

The accuracy of the proximity sensor 130 may depend at least on a heightof the UAV with respect to the external target. In some instances, theaccuracy of the proximity sensor 130 may be less than or equal to 0.1cm, 0.2 cm, 0.4 cm, 0.6 cm, 0.8 cm, 1 cm, 3 cm, 5 cm, 7 cm, 9 cm, 10 cm,12 cm, 15 cm, 17 cm, 20 cm, 23 cm, 25 cm, 27 cm, 30 cm, 33 cm, 35 cm, 37cm, 40 cm, 43 cm, 45 cm, 47 cm, 50 cm, 55 cm, 60 cm, 65 cm, 70 cm, 75cm, 80 cm, 85 cm, 90 cm, 95 cm, 100 cm, 110 cm, 120 cm, 130 cm, 140 cm,150 cm, 160 cm, 170 cm, 180 cm, 190 cm, 200 cm, 220 cm, 250 cm, or 300cm. Optionally, the accuracy of the proximity sensor 130 may be lessthan or equal to any of the values described herein. The proximitysensor 130 may have a accuracy falling within a range between any two ofthe values described herein.

The detection range of the proximity sensor 130 may depend onenvironmental factors of the UAV. In some instances, the minimumdistance in the detection range of the proximity sensor 130 may be equalto or more than 5 mm, 10 mm, 20 mm, 40 mm, 60 mm, 80 mm, 100 mm.Optionally, the minimum distance in the detection range of the proximitysensor 130 may be equal to or more than any of the values describedherein. The proximity sensor 130 may have a minimum distance in thedetection range falling within a range between any two of the valuesdescribed herein. In some instances, the maximum distance in thedetection range of the proximity sensor 130 may be equal to or more than1 m, 5 m, 10 m, 15 m, 20 m, 25 m, 30 m, 35 m, 40 m, 45 m, 50 cm, 55 m,60 m, 65 m, 70 m, 75 m, 80 m, 85 m, 90 m, 95 m, 100 m, 110 m, 120 m, 130m, 140 m, 150 m, 160 m, 170 m, 180 m, 190 m, 200 m, 300 m, 400 m, 500 m,600 m, 700 m, 800 m, 900 m, 1000 m, 1200 m, 1400 m, 1600 m, 1800 m, 2000m, 2500 m, 3000 m, 3500 m, 4000 m, 4500 m, 5000 m, 5500 m, or 6000 m.Optionally, the maximum distance in the detection range of the proximitysensor 130 may be equal to or more than any of the values describedherein. The proximity sensor 130 may have a maximum distance in thedetection range falling within a range between any two of the valuesdescribed herein.

In some embodiments, an initial positional relationship between themonocular camera unit 120 and the proximity sensor 130 may be measuredand determined in advance and thus may be predetermined. The positionalrelationship may include at least a distance between the monocularcamera unit 120 and the proximity sensor 130 and an angle of theproximity sensor 130 with respect to an optical axis of the monocularcamera unit 120. For example, if the proximity sensor 130 is a lidarsensor, the distance between the monocular camera unit 120 and the lidar130 may be a distance between the central lens point of the monocularcamera unit 120 and the central point of the laser emission element ofthe lidar 130. For example, if the proximity sensor 130 is a lidarsensor, the angle of the proximity sensor 130 with respect to an opticalaxis of the monocular camera unit 120 may be a relative angle of a laserray emitted from the lidar sensor with respect to an optical axis of themonocular camera.

In some instances, a distance between the monocular camera unit 120 andthe proximity sensor 130 may be less than or equal to 1 mm, 5 mm, 1 cm,3 cm, 5 cm, 10 cm, 12 cm, 15 cm, 20 cm, 25 cm, 30 cm, 35 cm, 40 cm, 45cm, 50 cm, 55 cm, 60 cm, 65 cm, 70 cm, 75 cm, 80 cm, 85 cm, 90 cm, 95cm, 100 cm, 110 cm, 120 cm, 130 cm, 140 cm, 150 cm, 160 cm, 170 cm, 180cm, 190 cm, 200 cm, 220 cm, 250 cm, or 300 cm. Optionally, the distancebetween the monocular camera unit 120 and the proximity sensor 130 maybe greater than or equal to any of the values described herein. Thedistance between the monocular camera unit 120 and the proximity sensor130 may have a value falling within a range between any two of thevalues described herein.

In some instances, an angle of the proximity sensor 130 with respect toan optical axis of the monocular camera unit 120 may be less than orequal to 1°, 5°, 10°, 15°, 20°, 25°, 30°, 35°, 40°, 45°, 50°, 55°, 60°,65°, 70°, 75°, 80°, 85°, 90°, 100°, 110°, 120°, 130°, 140°, 150°, 160°,170° or 180°. Optionally, the angle of the proximity sensor 130 withrespect to an optical axis of the monocular camera unit 120 may begreater than or equal to any of the values described herein. The angleof the proximity sensor 130 with respect to an optical axis of themonocular camera unit 120 may have a value falling within a rangebetween any two of the values described herein.

The monocular camera unit 120 and the proximity sensor 130 may bemounted at any suitable portion of the UAV. In some instances, themonocular camera unit 120 may be mounted on the body of the UAV, or apayload of the UAV, or an arm of the UAV. Meanwhile, the monocularcamera unit 120 may be mounted on the body of the UAV, or a payload ofthe UAV, or an arm of the UAV. In some instances, the monocular cameraunit 120 may be mounted on the proximity sensor 130. Alternatively, theproximity sensor 130 may be mounted on the monocular camera unit 120.

The positional relationship between the monocular camera unit 120 andthe proximity sensor 130 may be fixed or changed over time. In someinstances, both the monocular camera unit 120 and the proximity sensor130 may be fixed on the UAV during the flight of UAV. Alternatively, themonocular camera unit 120 may be fixed on the UAV while the proximitysensor 130 may be allowed to move rotationally or laterally relative tothe UAV during the flight of UAV. Alternatively, the proximity sensor130 may be fixed on the UAV while the monocular camera unit 120 may beallowed to move rotationally or laterally relative to the UAV during theflight of UAV. Optionally, both the monocular camera unit 120 and theproximity sensor 130 may be allowed to move rotationally or laterallyrelative to the UAV during the flight of UAV.

The positional relationship between the monocular camera unit 120 andthe proximity sensor 130 may be measured before the flight of UAV and/orduring the flight of UAV. The positional relationship may be measuredperiodically. In some instance, the positional relationship may bemeasured every 0.01 s, 0.02 s, 0.05 s, 0.07 s, 0.1 s, 0.2 s, 0.5 s, 0.7s, 1.0 s, 1.5 s, 2.0 s, 2.5 s, 3.0 s, 3.5 s, 4.0 s, 4.5 s, 5.0 s, 5.5 s,6.0 s, 6.5 s, 7.0 s, 7.5 s, 8.0 s, 8.5 s, 9.0 s, 9.5 s, or 10.0 s. Thecurrent positional relationship may be calculated based on a previousmeasurement of the positional relationship and a relative movement ofthe UAV since the previous measurement, with aid of one or moreprocessors onboard or off-board the UAV.

In some instances, the predetermined positional relationship between themonocular camera unit 120 and the proximity sensor 130 may be stored ina memory unit onboard the UAV. Alternatively, the predeterminedpositional relationship may be received (e.g., using a wired or wirelessconnection) from a remote terminal before or during the flight.

The UAV 110 may carry a processing unit (not shown in FIG. 1) onboard.The processing unit may comprise a flight controller which controls anoperation of the UAV 110. In some embodiments, the processing unit mayfurther comprise one or more processors. In some instances, theprocessing unit may be configured to measure the positionalrelationship, estimate the external state information of the UAV fromcaptured images, analyze the proximity data for the external objects todetermine the relative proportional relationship, and/or apply therelative proportional relationship to the determined external stateinformation. Alternatively, any of the measuring, estimating, analyzingand applying procedures may be performed by an off-board processingunit, for example, a remote terminal. Optionally, the measuring,estimating, analyzing and applying procedures may be performed by acombination of processing unit onboard the UAV and processing unitoff-board the UAV.

In some instances, the environment of UAV 110 flight may be a relativelysimple environment. The simple environment may be a natural environmentor an artificial environment. A simple environment may be, for example,a clear open outdoor, a suburban area with less buildings, or an indoorplace with few obstacles. Alternatively, the environment of UAV flightmay be a relative complex environment. The relative complex environmentmay be a natural environment or an artificial environment. A relativecomplex environment may be, for example, a mountainous region withcomplex terrain, or an urban area with tall buildings. The UAV may beconfigured to move from a relatively simple environment to a relativelycomplex environment or vice versa. Based on the environment, the UAV maybe configured to execute certain autonomous navigation routines. Forexample, the UAV may be configured to execute autonomous obstacle orcollision avoidance routines so as to detect and avoid obstacles in itssurrounding environment.

External objects include objects in the UAV's external environment.Examples of the external objects may include, but are not limited to,buildings, mountains, trees, roads, or vehicles. In some instances, theexternal object may be beneath, above or next to the UAV while the UAVis in flight. The external object may remain at a stationary heightrelative to an underlying surface of the environment, such as a groundor structure. The external object may be a stationary object on theground, for example, a tree or a building. Alternatively, the externalobject may be a movable object, for example, a vehicle.

A distance from the UAV to external object may vary in differentapplication fields. In some instances, the UAV may be away from theexternal object by a distance more than or equal to 1 cm, 3 cm, 5 cm, 8cm, 10 cm, 15 cm, 20 cm, 25 cm, 30 cm, 35 cm, 40 cm, 45 cm, 50 cm, 60cm, 70 cm, 80 cm, 90 cm, 1 m, 1.3 m, 1.5 m, 1.8 m, 2 m, 2.5 m, 3 m, 3.5m, 4 m, 4.5 m, 5 m, 6 m, 7 m, 8 m, 9 m, 10 m, 13 m, 15 m, 18 m, 20 m, 23m, 25 m, 28 m, 30 m, 35 m, 40 m, 45 m, 50 m, 55 m, 60 m, 65 m, 70 m, 80m, 90 m, 100 m, 150 m, 200 m, 250 m, or 300 m. Optionally, the distancebetween the UAV and the external object may be greater than or equal toany of the values described herein. The distance between the UAV and theexternal object may have a greatest dimension falling within a rangebetween any two of the values described herein.

FIG. 2 illustrates a process 200 for determining and updating a state ofUAV in accordance with some embodiments. Aspects of the process 200 maybe performed, individually or collectively, by one or more processorsonboard the UAV, one or more processors off-board the UAV, or anycombination thereof. Some or all aspects of the process 200 (or anyother processes described herein, or variations and/or combinationsthereof) may be performed under the control of one or morecomputer/control systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. The order in which the operations aredescribed is not intended to be construed as a limitation, and anynumber of the described operations may be combined in any order and/orin parallel to implement the processes.

In step 201, external state information of the UAV may be determinedbased on image data of an environment of the UAV captured by themonocular camera unit onboard the UAV.

The external state information of the UAV may be the information of UAVrelative to external objects. In some embodiments, the external stateinformation of the UAV may include distance information relative to oneor more external objects. In some embodiments, the external stateinformation of the UAV may include position information of one or moreexternal objects within the environment or position information of theUAV. In other embodiments, the external state information of the UAV maycomprise a plurality of feature points, each associated with a first setof coordinates under a local reference frame. Since the monocular cameraunit is mounted on the UAV, the external state information of the UAVmay be equivalent to or otherwise derived from the external state of themonocular camera. In some embodiments, the external state of the UAV maynot be equivalent to or otherwise derived from the external state of themonocular camera. In some embodiment, the external state information ofthe UAV, i.e., the external information of the monocular camera, may beacquired by analyzing feature points on the images captured by themonocular camera.

Although in step 201 the external state information of the UAV isdescribed as estimated from images acquired from the monocular camera,the external state information of the UAV may be obtained from a stereoor multiple cameras. In some embodiments, for example, a pair of visionsensors (e.g., binocular cameras) may be used to estimate the externalstate of the UAV. In some embodiments, the pair of vision sensors arelaterally spaced apart on the UAV such that each vision sensor providesan image from a different camera viewpoint, thereby enabling stereovision imaging. For example, the pair of vision sensors may be separatedlaterally by up to 1 m, 500 cm, 250 cm, 100 cm, 50 cm, 25 cm, 10 cm, 5cm, 2 cm, or 1 cm. The vision sensors can be disposed on the same sideof the UAV or opposite sides of the UAV. One or more vision sensors canbe disposed on the front, rear, top, bottom, or lateral sides of theUAV, or suitable combinations thereof.

In some embodiments, raw image data obtained by the camera can beprocessed by image analysis to correct the image data, for example, byimplementing an algorithm to correct optical distortion and/or noise inthe image data. The image analysis can extract suitable feature pointsfrom the corrected image data and generate a digital representation ofthe feature points, as previously described herein.

In some embodiments, frame-to-frame matching of the feature pointsproduced by the image analysis can be performed using an initialestimation of the positional change, thereby reducing the complexity ofthe search and improving matching efficiency and accuracy. For example,the initial estimation can be generated based on two-way sensor fusionof the IMU and the GPS sensor onboard the UAV. The IMU can providevelocity data with respect to six degrees of freedom that can becompared with positional information from the previous frame to providea measurement of positional change ({tilde over (R)}_(I), {tilde over(T)}_(I)). The GPS sensor can provide measurement data of the positionalchange of the movable object, {tilde over (T)}_(G) (e.g., with respectto three degrees of translation). In some embodiments, the GPSmeasurement data can be assigned a weight value δ, as described herein.The weight value δ can range from 0 to 1 based on the strength of theGPS signal. For example, in an environment where the GPS signal is weak,δ can be close to 0. In an environment where the GPS signal is strong, δcan be close to 1. Examples of an environment where the GPS signal canbe weak can include an indoor environment, obstructions by structures ornaturally occurring features, inclement weather conditions, lack ofsatellites overhead, or malfunction with a GPS receiver. Examples ofenvironments where the GPS signal can be strong can include an outdoorenvironment, high altitude, lack of obstructions, or clear weatherconditions. In one scenario, a movable object may be flying at lowaltitude surrounded by a number of tall buildings. This may result inblocking or weakening satellite signals, which may correspondingly causethe data from the GPS system to be accorded lower weight. This mayresult in data from the GPS system being discounted relative to othersensing systems that have no indication of being compromised.Accordingly, the IMU data ({tilde over (R)}_(I), {tilde over (T)}_(I))and the GPS sensor data (δ, T_(G)) can be fused using any suitablesensor fusion method as described herein (e.g., a Kalman filter) togenerate an initial estimation of the positional change of the movableobject ({tilde over (R)}, {tilde over (T)}).

The initial estimation ({tilde over (R)}, {tilde over (T)}) can be usedto search for a feature point in a current frame, m_(t) ^(i), (in whichi represents the i^(th) feature point and t represents the current time)corresponding to the same feature point in a previous frame, m_(t-1)^(i). For example, the estimated 3D coordinate {circumflex over(p)}_(t-1) ^(i)=({circumflex over (x)}_(t-1) ^(i), ŷ_(t-1) ^(i),{circumflex over (z)}_(t-1) ^(i)) of the feature point m_(t-1) ^(i) canbe rotated according to ({tilde over (R)}, {tilde over (T)}) to obtain{tilde over (R)}. {circumflex over (p)}_(t-1) ^(i)+{tilde over (T)}. Theestimated 3D coordinate {circumflex over (m)}_(t) ^(i)=KΠ₀({tilde over(R)}·{circumflex over (p)}_(t-1) ^(i)+{tilde over (T)}) can be projectedto obtain the position of the feature point after the rotation{circumflex over (m)}_(t) ^(i)=(û_(t) ^(i), {circumflex over (v)}_(t)^(i)), where K is the internal parameter matrix for the camera and Π₀ isthe projection matrix. The current frame can then be searched andmatched for the feature point within a (û_(t) ^(i), {circumflex over(v)}_(t) ^(i)) area, with a point m_(t) ^(i)=(u_(t) ^(i), v_(t) ^(i))having the minimal planar distance from (û_(t) ^(i), {circumflex over(v)}_(t) ^(i)) considered to be the corresponding point. Accordingly,the frame-to-frame relationship between the feature points m_(t-1) ^(i)and m_(t) ^(i) can be determined.

The determined/estimated external state information of the UAV in step201 may be inaccurate due to an accumulation of errors over time. Forexample, a deviation between the estimated external state informationand the actual state information may become larger as time elapses, sothat a reliability of the visual navigation is reduced. Therefore, theexternal state estimated from the monocular camera unit may need to beupdated using data from a different sensor in order to improve theaccuracy. For example, the external state estimated from the monocularcamera unit may be updated using external state information measured bya different type of sensor. The different sensor for updating theestimated external state may be any type of sensor other than amonocular camera, not limited to a proximity sensor as described hereinas an embodiment. In some instances, the different sensor may be a GPSsensor, a motion sensor, an inertial sensor, or an image sensor.

In step 202, a relative proportional relationship may be calculated. Therelative proportional relationship may be applied to the determinedexternal state information of the UAV which is obtained in step 201. Therelative proportional relationship may be calculated based at least inpart on: (1) a predetermined positional relationship between themonocular camera unit and a proximity sensor and (2) proximity dataabout the environment of the UAV that is acquired by the proximitysensor.

In some embodiments, the proximity data may include a distance to anexternal object which is measured by the proximity sensor.

In some embodiments, the relative proportional relationship may includea scale factor. The scale factor may include a number or atransformation function that can be used to convert coordinates under afirst reference frame to coordinates under a second reference frame, asdiscussed in further detail in FIG. 3. In some instances, the firstreference frame may be a local reference frame, for example, an imagereference frame, and the second reference frame may be a globalreference frame. In some embodiments, the proximity sensor may includeat least one of a lidar sensor, an ultrasound sensor, or an infraredsensor. A detailed discussion on a calculation of the relativeproportional relationship will be provided in FIG. 3.

In step 203, the determined external state information of the UAV ofstep 201 may be updated, individually or collectively by the one or moreprocessors onboard or off-board the UAV, at least by applying thecalculated relative proportional relationship of step 202 to thedetermined external state information. For example, the determinedexternal state information of UAV may be multiplied by the relativeproportional relationship.

In some embodiments, not all the feature points on an image captured bythe monocular camera unit or pair of vision sensors may be detected bythe proximity sensor. The relative proportional relationship may becalculated from those feature points which are detectable by theproximity sensor, and then applied to those feature points which are notdetectable by the proximity sensor, as will be provided in FIG. 3.

By performing the process described in FIG. 2, the updated externalstate information of UAV may be obtained, which is more precise or evenequal to the actual state information of the UAV. In some embodiments,the estimated external state information may be updated periodically. Insome instance, the estimated external state information may be updatedevery 0.01 s, 0.02 s, 0.05 s, 0.07 s, 0.1 s, 0.2 s, 0.5 s, 0.7 s, 1.0 s,1.5 s, 2.0 s, 2.5 s, 3.0 s, 3.5 s, 4.0 s, 4.5 s, 5.0 s, 5.5 s, 6.0 s,6.5 s, 7.0 s, 7.5 s, 8.0 s, 8.5 s, 9.0 s, 9.5 s, or 10.0 s.

FIG. 3 illustrates an exemplary calculation 300 of a relativeproportional relationship which is to be applied on the determinedexternal state information of the UAV, in accordance with an embodimentof the disclosure.

In the embodiment of FIG. 3, the external state information may includea plurality of feature points, each associated with a first set ofcoordinates under a first reference frame. The feature points withineach image captured by the monocular camera unit may include, but notlimited to, an edge of an object, a corner of an object, or a boundarybetween objects of two different colors. Any suitable method orcombination of methods may be used to identify and provide a digitalrepresentation of the feature points, as discussed herein above. Theimage data may then be frame-to-frame matched to each other to identifya set of common feature points appearing in the series of image data.The external state information of the UAV may be determined based on thecommon feature points and a time interval between two images, asdiscussed above.

In the embodiment of FIG. 3, the relative proportional relationship maybe a scale factor. This scale factor may be a scale between the secondreference frame and the first reference frame. The first reference framemay be a local reference frame, for example, an image reference frame.The second reference frame may be a global reference frame.

The X-Y coordinate system of FIG. 3 may be a local reference frame, forexample, an image reference frame or a camera reference frame in whichthe camera is located at the origin. As shown in FIG. 3, the location ofthe monocular camera unit C may be set as the origin of the X-Ycoordinates, and a lidar sensor L may be used as the proximity sensor. Apositional relationship between the monocular camera unit C and thelidar sensor L may be predetermined. In some instances, a relativedistance H and a relative angle θ of a laser ray emitted from the lidarsensor L with respect to an optical axis of the monocular camera unit Cmay be measured and predetermined in advance. Alternatively, the arelative distance H and a relative angle θ of a laser ray emitted fromthe lidar sensor L with respect to an optical axis of the monocularcamera unit C may be determined during the flight of UAV. Optionally, arelative distance H and a relative angle θ of a laser ray emitted fromthe lidar sensor L with respect to an optical axis of the monocularcamera unit C may be calculated based on a previous measurement of thepositional.

In some instances, determining external state information of the UAV maycomprise extracting the plurality of feature points from the image data;and determining the first sets of coordinates respectively associatedwith the plurality of feature points under the first reference frame.

In some instances, the proximity data may include proximity informationfor a first feature point of the plurality of feature points and doesnot include proximity information for a second feature point of theplurality of feature points.

In a set of feature points [W1, W2, W3, Wn] on an image captured by themonocular camera unit C, some of the feature points may be detected andmeasured by the lidar sensor L while others may not. Accordingly, theset of feature points [W1, W2, W3, Wn] may be classified into twosubsets: a first subset of proximity sensor detectable feature points,and a second subset of proximity sensor non-detectable feature points.For example, the lidar sensor L may detect feature points W1 and W2because they are in the line of the emitted laser, but may not detectfeature points W3 and W4 because they are out of the line of the emittedlaser. In the embodiment of FIG. 3, the lidar sensor L may measure adistance to a detectable feature point W1 as S1. Since the UAV and/orlidar sensor L can move, the lidar sensor L may detect different pointsat different times. For example, the lidar sensor L may detect featurepoints W1 and W2 the first time; but may detect feature points W3 thesecond time if the position/direction of the lidar sensor L rotates.

In some instances, calculating the relative proportional relationshipmay include: determining a second set of coordinates under a secondreference frame for the first feature point based on (1) the proximityinformation for the first feature point and (2) the predeterminedpositional relationship between the monocular camera unit and theproximity sensor unit; and using the first set of coordinates and thesecond set of coordinates for the first feature point to determine therelative proportional relationship. The first reference frame may be alocal reference frame, for example, an image reference frame or camerareference frame. The second reference frame may be a global referenceframe.

Regardless of being detectable or non-detectable by the lidar sensor,the coordinate cn (xn, yn) associated with any one of feature point Wnunder the local reference frame (e.g., the image reference frame) may bemeasured and determined by any known methods, such as monocular SLAM,which is a method for determining a coordinate of arbitrary featurebased on single camera. In some instances, the coordinate associatedwith the lidar sensor detectable feature point W1 under the localreference frame may be measured as c1 (x1, y1). The coordinate C1 (X1,Y1) of the same feature point W1 under the global reference frame maythen be calculated based on (1) the predetermined relative distance H,the predetermined relative angle θ and (2) the measured distance S1 as:

X1=H×cos θ−S1×sin θ

Y1=H×sin θ+S1×cos θ

Here, a scale factor λ for updating the determined external stateinformation of the UAV may be calculated by:

λ=C1/c1

In some embodiments, the scale factor λ may be determined based onmultiple points. For example, the scale factor λ may be an average valueof a number of calculated scale factors that are calculated based on anumber of points in accordance with the above method.

The calculated scale factor λ may then be used to update the determinedexternal state information of UAV. For example, as discussed hereinabove, the scale factor λ, which is calculated from those feature pointswhich are detectable by the proximity sensor, may be applied to updatethose feature points which are not detectable by the proximity sensor.

In some embodiments, the calculated scale factor λ may be applied toupdate the distance information of the UAV relative to one or moreexternal objects within the environment. In other embodiments, thecalculated scale factor λ may be applied to update the positioninformation of one or more external objects within the environment orposition information of the UAV. For example, the calculated scalefactor λ may be applied to update the world coordinates of one or moreexternal objects or world coordinates of the UAV. In other embodiments,the calculated scale factor λ may be applied to update the coordinatesof the feature points. In some instances, the scale factor λ may be usedto transform or translate coordinates under a first coordinate system(e.g., in pixels) to coordinates under a second coordinate system (e.g.,in meters).

In some embodiments, updating the external state information may includeapplying the relative proportional relationship to the first set ofcoordinates associated with at least one of the plurality of featurepoints under a first reference frame to obtain a second set ofcoordinates under a second reference frame.

For example, in the embodiment of FIG. 3, the calculated scale factor λmay be a scale between the global reference frame and the localreference frame. By applying the calculated scale factor λ to a localcoordinate cn (xn, yn) of a feature point Wn in the set of featurepoints [W1, W2, W3, Wn], the global coordinate Cn (Xn, Yn) of thisfeature point Wn may be calculated by:

Cn=λ*cn

In some instances, the coordinate of a proximity sensor non-detectablefeature point Wn under a second reference frame may be calculated byapplying the calculated scale factor λ to the coordinate of Wn under afirst reference frame which is already known.

In some instances, the coordinate of a proximity sensor detectablefeature point Wn under a second reference frame may also be calculatedby applying the calculated scale factor λ to the coordinate of Wn undera first reference frame.

In other instances, the coordinate of any feature point Wn under asecond reference frame may also be calculated by applying the calculatedscale factor λ to the coordinate of Wn under a first reference frame,regardless of being detectable or non-detectable by the proximitysensor.

In some instances, the position and/or orientation of the camera and/orthe proximity sensor may change during the UAV's flight. Therefore, thevalue of the scale factor λ may change over time as a result of a changein the positional relationship between the camera and the proximitysensor. For example, the monocular camera unit may rotate about a pitchaxis, roll axis, and/or yaw axis thereof. A rotating of the camera mayat least lead to a change in a relative angle θ of a laser ray emittedfrom the lidar sensor with respect to an optical axis of the monocularcamera. For another example, the orientation of the lidar sensor maychange during the UAV's flight, which may also lead to a change in thediscussed relative angle θ. For another example, the scale factor λ mayneed to be re-estimated if the monocular SLAM fails. Therefore, thescale factor λ may be calculated and updated each time a lidar sensordetectable feature point is detected by the lidar sensor, such that thescale factor is up to date. In some embodiments, the scale factor λ maybe calculated and updated at a regular interval. The regular intervalfor updating the scale factor λ may be 0.01 s, 0.02 s, 0.05 s, 0.07 s,0.1 s, 0.2 s, 0.5 s, 0.7 s, 1.0 s, 1.5 s, 2.0 s, 2.5 s, 3.0 s, 3.5 s,4.0 s, 4.5 s, 5.0 s, 5.5 s, 6.0 s, 6.5 s, 7.0 s, 7.5 s, 8.0 s, 8.5 s,9.0 s, 9.5 s, or 10.0 s. Optionally, the regular interval for updatingthe scale factor λ may be greater than or equal to any of the valuesdescribed herein.

FIG. 4 illustrates a process 400 for determining an external state ofUAV based on image data, in accordance with an embodiment of thedisclosure.

In step 401, historical external state information of the UAV may beobtained. The external information may comprise distance informationrelative to an external object, while the internal information mayinclude one or more of acceleration information, direction information,angular speed information, speed information or mileage information. Insome instances, an external sensor such as a laser sensor or anultrasonic sensor may be employed to measure the distance relative tothe external object, and an internal sensor such as an accelerationsensor, direction sensor or a mileage sensor may be employed to measureinternal information such as acceleration, direction or mileage.

In some embodiments, the historical state information of the UAV mayinclude, but not limited to, distance information of the UAV relative toone or more external objects within the environment, positioninformation of one or more external objects within the environment orposition information of the UAV, and/or a plurality of feature pointeach associated with a first set of coordinates under a first referencemodel.

In step 402, image data comprising a current image and an historicalimage preceding the current image may be obtained.

In some instances, the monocular camera unit may capture a series ofimages of an environment of the UAV over time. In this case, thehistorical image may be an image captured before a predetermined timeunit with respect to current image, from among a series of image frames.This predetermined time unit may be 0.1 s, 0.3 s, 0.5 s, 0.8 s, 1 s, 2s, 3 s, 4 s, 5 s, 6 s, 7 s, 8 s, 9 s, 10 s, 11 s, 12 s, 13 s, 14 s, 15s, 16 s, 17 s, 18 s, 19 s, 20 s, 23 s, 25 s, 28 s, 30 s, 35 s, 40 s, or60 s. The predetermined time unit may have a value within a rangebetween any two of the values described herein.

The current image may be obtained in real time from the monocularcamera. In some instances, the historical image preceding the currentimage may be read from a memory onboard the UAV which stores thecaptured images. The memory for storing the captured images may be anon-transitory computer readable medium, which may include one or morememory units (e.g., removable media or external storage such as an SDcard or random access memory (RAM)). Alternatively, the historical imagemay be received from a remote terminal, for example a remote datastorage or a remote computing device.

In step 403, a matching feature point in the current image correspondingto a target feature point in the historical image may be predicted,according to the historical state information.

The historical image and current image captured by the monocular cameraunit may include information of many external objects such as trees andbuildings, and any edge corner point of these objects may be used as atarget feature point. The historical external state information mayinclude information characterizing a motion vector such as direction andspeed of the UAV, therefore, the matching feature point in currentimage, which corresponds to the target feature point in the historicalimage, may be predicted according to the motion vector information.

In some instances, predicting the matching feature point in currentimage may include: predicting a coordinate position of the targetfeature point of the historical image in the current image according tothe historical state information; and selecting a feature point that isclosest to the predicted coordinate position as the matching featurepoint in the current image.

In step 404, current external state information of the UAV may becalculated, based on the matching feature point.

In some embodiments, a Kalman filter may be used in this step tocalculate the current external state information of the UAV. In otherembodiments, a measurement model may be incorporated in this step tocalculate the current external state information of the UAV, as will bediscussed hereinafter.

In some embodiments, the external information of the UAV may becalculated from the matching feature point. For example, a distance ofan external object relative to the UAV may be obtained by calculating adistance from the UAV to the matching feature point by using thecoordinate of the matching feature point.

Although the embodiment of FIG. 4 is discussed as calculating externalinformation of UAV based on historical external state information, theinternal information of the UAV may also be calculated from the matchingfeature point. For example, speed information of the UAV may becalculated based on the coordinate of the target feature point, thecoordinate of the matching feature point and the interval of timebetween the current image and the historical image.

The calculated external information of the UAV may then be updated byapplying a relative proportional relationship, which may be calculatedby a process as described in step 202 of FIG. 2 and particularly theprocess as described above with reference to FIG. 3.

In the embodiment of FIG. 4, the external state information of the UAVmay be determined based on image data of an environment of the UAVcaptured by the monocular camera unit onboard the UAV. However, thepresent disclosure may determine the external state information of theUAV based on visual data acquired from other types of optical sensors inconnection with data fusion techniques.

FIG. 5 is a schematic illustration by way of block diagram of a system500 for determining an external state of UAV in accordance with anembodiment of the present disclosure. The system 500 for determining anexternal state of UAV in accordance with an embodiment of the presentdisclosure may include a capturing unit 501, a matching unit 502, arelative proportional relationship computing unit 503 and a updatingunit 504. The capturing unit 501 may be coupled to the matching unit 502and the relative proportional relationship computing unit 503. Thematching unit 502 and the relative proportional relationship computingunit 503 may be coupled to the updating unit 504. In an embodiment, thecapturing unit 501 may for example comprise a monocular camera unit 5011and a Lidar sensor 5012.

The operating of the system 500 may be performed with the aid of one ormore processors. The processor may be provided as part of controlcircuit of the UAV, or may be implemented by the flight controller ofthe UAV, or may be provided as an independent circuit, module or chip.The processor may be implemented by Central Processing Unit (CPU),Application Specific Integrated Circuit (ASIC), or Field ProgrammableGate Array (FPGA). Any description herein of a processor may apply toone or more processors, which may individually or collectively performany functions described for the processor. The processor may be capableof executing one or more steps in accordance with non-transitorycomputer readable media comprising code, logic, or instructions forperforming one or more steps. Memory storage units may be provided whichmay comprise the non-transitory computer readable media.

One or more processors may be provided onboard the UAV. The signals andinformation may be processed onboard the UAV. One or more processors maybe provided off-board the UAV. The signals and information may beprocessed off-board the UAV. In some instances, one or more processorsmay be distributed over the UAV and one or more external devices, orover a plurality of external devices. The processors that may bedistributed over the UAV and/or devices may individually or collectivelygenerate the processed signals.

The capturing unit 501 may obtain historical external state informationof the UAV, a current image acquired by the monocular camera unit and anhistorical image preceding the current image. The capturing unit 501 mayalso obtain a proximity data from the lidar sensor 5012. For example,the proximity data is a distance between the lidar sensor 5012 and anobject external to the UAV, as discussed herein above. The capturingunit 501 may also obtain a positional relationship between the monocularcamera unit 5011 and the lidar sensor 5012, which for example is arelative distance H and a relative angle θ of a laser ray emitted fromthe lidar sensor 5012 with respect to an optical axis of the monocularcamera unit 5011, as discussed herein above.

The external state information of UAV at the time of taking off may beinitialized (e.g., set to zero), and the external state information ateach time unit after UAV's taking off may be related to the externalstate information at the previous time unit. Since the monocular cameraunit is fixed to the UAV, the external state information of the UAV maybe considered as the state information of the monocular camera. In someinstances, the external state information may at least comprise theposture, position, speed, direction, coordinates and surroundingenvironmental information of the UAV.

The historical external state information of the UAV, current imageacquired by the monocular camera unit and historical image preceding thecurrent image may then be provided to the matching unit 502. Thematching unit 502 may be configured to: (1) according to the historicalstate information, predict a matching feature point in the current imagecorresponding to a target feature point in the historical image; and (2)estimate, from the matching feature point, current external informationof the UAV. The determined current external information of the UAV maybe outputted to the updating unit 504 for further processing.

The image captured by the monocular camera unit 5011 may includeinformation of many external objects such as trees and buildings, andany edge corner point of these objects may be used as a target featurepoint. The historical external state information may include informationcharacterizing a motion vector such as direction and speed of the UAV,therefore, the matching feature point in current image, whichcorresponds to the target feature point in the historical image, may bepredicted according to the motion vector information.

In some instances, the matching unit 502 may predict a coordinateposition in the current image according to the historical stateinformation and the target feature point in the historical image, andselect, as the matching feature point in current image, a feature pointthat is closest to the predicted coordinate position.

In some instances, a displacement of the coordinate position of thetarget feature point may be obtained by using the historical externalstate information so as to predict the coordinate position of the targetfeature point in the current image. This process may be implemented byfeature matching algorithms as discussed hereinabove, including but notlimited to an optical flow method, or a matching method based on featuredescriptors.

The external information of the UAV may be calculated from the matchingfeature point. For example, a distance of an external object relative tothe UAV may be obtained by calculating a distance from the UAV to thematching feature point by using the coordinate value of the currentfeature point. For another example, speed information of the UAV may becalculated based on the coordinate value of the current feature point,the coordinate value of the target feature point and the interval oftime unit.

The relative proportional relationship computing unit 503 may calculatea relative proportional relationship, based at least in part on: (1) apredetermined positional relationship between the monocular camera unit5011 and the lidar sensor 5012, which is for example a relative distanceH and a relative angle θ of a laser ray emitted from the lidar sensor5012 with respect to an optical axis of the monocular camera unit 5011,as discussed herein; and (2) a proximity data from the lidar sensor5012, which for example is a distance between the lidar sensor 5012 andan object external to the UAV, as discussed herein above, which areinput from the capturing unit 501. This relative proportionalrelationship may be applied to update the determined external stateinformation, which is the output of the matching unit 502.

The calculating process for the relative proportional relationship maybe substantially identical to that described with reference to FIG. 3.The calculated relative proportional relationship, which is the outputof the relative proportional relationship computing unit 503, and thedetermined external state information of UAV, which is the output of thematching unit 502, may then be fed to the updating unit 504.

The updating unit 504 may update the determined external stateinformation of UAV by applying the relative proportional relationship.For example, the updating unit 504 may be configured to multiply thedetermined external state information of UAV by the relativeproportional relationship. To this end, the updated external stateinformation of UAV is obtained, which is more precise or even equal tothe actual state information of the UAV.

FIG. 6 is a flow chart illustrating a method 600 of determining andupdating a state of UAV by applying Kalman filter, in accordance with anembodiment of the disclosure. The Kalman filter is an algorithm thatuses a series of measurements observed over time, containing noise(random variations) and other inaccuracies, and produces estimates ofunknown variables that tend to be more precise than those based on asingle measurement alone. In some embodiments, Kalman filter may be usedto estimate the current state information of a UAV. Alternatively,Extended Kalman filter (EKF) may be used. In the embodiment of FIG. 6,the current external of UAV may be obtained by applying a historicalexternal state information to a pre-established state model and apre-established measurement model.

In some embodiments, state model may indicate how the state propagate tothe next timestamp, for example:

$\begin{pmatrix}v_{t + 1} \\a_{t + 1}\end{pmatrix} = \begin{pmatrix}{v + {a_{t}t}} \\a_{t}\end{pmatrix}$

wherein v is a velocity, a is an acceleration, and

$\quad\begin{pmatrix}v \\a\end{pmatrix}$

is a state.

In some embodiments, measurement model may indicate how the state ismeasured.

In step 601, the UAV may be Initialized, and a predetermined positionalinformation between the camera unit and at least one kind of othersensor, a pre-established state model, a pre-established measurementmodel, a pre-established camera calibrating model and the initial stateinformation of the camera unit may be obtained.

In some embodiments, the predetermined positional information betweenthe camera unit and at least one kind of other sensor may be similar tothat discussed in step 202 of FIG. 2 and that in FIG. 3, for example, apredetermined positional relationship between a monocular camera unitand a proximity sensor onboard the UAV may be obtained.

The pre-established state model, pre-established measurement model,pre-established camera calibrating model may be established in advance.

In some embodiments, an equation of the state model may be:

x(k|k−1)=Ax(k−1|k−1)+BU(k)+w(k)

In the equation of state model, k is a time coefficient, x(k|k−1) is apredicted external state information, x(k−1|k−1) is an estimatedprevious external state information, U(k) is a known input quantity, wis a process noise, and A and B are state parameters.

In some embodiments, an equation of the measurement model may be:

Z(k)=Hx(k|k−1)+v(k)

In the an equation of measurement model, Z(k) is a measurement of thematching feature point, v is process noise, and H is a measurementparameter.

In step 602, a predicted external state information of the UAV may bepredicted according to a previous state information and the state model.

In step 603, a current image taken by the camera unit and a previousimage preceding the current image may be acquired, and feature pointsmeasured by a lidar sensor under global reference frame may be obtained.

In step 604, the current image may be calibrated according to thepre-established calibrating model. An example of an uncalibrated andcalibrated image is provided in FIG. 8 and FIG. 9, respectively.

In step 605, a matching feature point in the calibrated current imagemay be predicted according to previous state information; and ameasurement of matching feature point may be obtained according to themeasurement model and the predicted matching feature point. The processof predicting a matching feature point may be identical to steps 401-403as discussed in FIG. 4, as discussed herein above. In the equation ofthe measurement model, x(k|k−1) is the target feature point, and is oneparameter in the predicted external state information of the UAV.

In step 606, the predicted external state information of the UAV and themeasurement of matching feature point may be filtered by a Kalman filterto obtain estimated current state information of the UAV. The timeupdating equation of the Kalman filter may be:

x(k|k)=x(k|k−1)+G(k)(Z(k)−Hx(k|k−1))

G(k)=P(k|k−1)H′/(HP(k|k−1)H′+R)

In the above time updating equation of the Kalman filter, x(k|k) is anestimated external state information, G(k) is a Kalman gain, P(k|k−1) isa covariance corresponding to x(k|k−1), R is a covariance of v, and H′is a transposed matrix of H.

In step 607, a relative proportional relationship may be calculatedbased on a first distance from the lidar sensor to a feature point underglobal reference frame and a second distance from the camera unit to thematching feature point under a local reference frame. The process ofcalculating the relative proportional relationship in step 607 may besimilar to those discussed in step 202 in FIG. 2 and those discussed inFIG. 3.

In step 608, the estimated current external state information of the UAVand the a relative proportional relationship may be subjected to aproduct operation, such that the estimated current external stateinformation may be updated to obtain an updated external stateinformation of UAV, which may be more precise or even equal to theactual state information of the UAV. This updating process in step 608may be similar to that discussed in step 203 of FIG. 2.

In some embodiments, the determined external state information of theUAV may be fed to other types of filters, including but not limited toan Extended Kalman filter.

FIG. 7 is a schematic illustration by way of block diagram of a system700 for determining and updating a state of UAV in accordance with anembodiment of the present disclosure. Components of the system may beonboard/offboard the UAV.

As shown in FIG. 7, the system 700 for determining a state of UAV inaccordance with an embodiment of the present disclosure may include acapturing unit 701, a matching unit 702, a relative proportionalrelationship computing unit 703 and an updating unit 704. The capturingunit 701 may be coupled to the matching unit 702 and the relativeproportional relationship computing unit 703. The matching unit 702 andthe relative proportional relationship computing unit 703 may be coupledto the updating unit 704. In an embodiment, the capturing unit 701 mayalso comprise a monocular camera unit 7011 and a Lidar sensor 7012.

The operation of the capturing unit 701, the matching unit 702 and therelative proportional relationship computing unit 703 in system 700 ofthe embodiment may be substantially identical to the capturing unit 501,matching unit 502 and the relative proportional relationship computingunit in 503 in system 500 of the embodiment shown in FIG. 5, asdiscussed herein above. The functionalities of the system 700 may beperformed with the aid of one or processors onboard or off-board. Thecalculated relative proportional relationship, which is the output ofthe relative proportional relationship computing unit 703, and thedetermined external state information of UAV, which is the output of thematching unit 702, may be fed to the updating unit 704.

The updating unit 704 of this embodiment may comprise a filtering unit7041 and relative proportional relationship applying unit 7042 which areinterconnected. The filtering unit 7041 may be configured to apply aKalman filter to the historical external state information of the UAVaccording to a pre-established state model and a pre-establishedmeasurement model, to obtain an estimated current external stateinformation of UAV. Then, the estimated current external stateinformation of UAV may be fed to the relative proportional relationshipapplying unit 7042, which may be configured to update the estimatedcurrent external state information of UAV by applying the relativeproportional relationship. For example, the relative proportionalrelationship applying unit 7042 may be configured to multiply theestimated current external state information of UAV by the relativeproportional relationship. To this end, the updated external stateinformation of UAV may be obtained, which is more precise or even equalto the actual state information of the UAV.

In some instances, the optimized external state information of the UAVmay be calculated by applying other types of filters, including but notlimited to an Extended Kalman filter.

FIG. 8 shows an original image 800 captured by the monocular camera,which is a panoramic type, as an example of image calibration inaccordance with an embodiment of the present disclosure. FIG. 9 shows acalibrated image 900 after column-expansion calibration as an example ofimage calibration in accordance with an embodiment of the presentdisclosure.

The calibration of panoramic cameras may be similar to that forcalibrating standard perspective cameras. The calibration methods ofpanoramic cameras may take advantage of planar grids that are shown bythe user at different positions and orientations. For omnidirectionalcameras, it is very important that the calibration images are taken allaround the camera and not on a single side only. This aims to compensatefor possible misalignments between the camera and mirror.

Three open-source calibration toolboxes may be currently available forMatlab, which differ mainly for the projection model adopted and thetype of calibration pattern. The first calibration toolbox is Mei, whichuses checkerboard-like images and takes advantage of the projectionmodel of Geyer and Daniilidis. It may be particularly suitable forcatadioptric cameras using hyperbolic, parabolic, folded mirrors, andspherical mirrors. The second calibration toolbox is Barreto, which usesline images instead of checkerboards. Like the toolbox of Mei, it mayalso use the projection model of Geyer and Daniilidis. It may beparticularly suitable for parabolic mirrors. Finally, the toolbox ofScaramuzza uses checkerboard-like images. Contrary to the previous two,it takes advantage of the unified Taylor model for catadioptric andfisheye cameras. It may work with catadioptric cameras using hyperbolic,parabolic, folded mirrors, spherical, and elliptical mirrors.Additionally, it may work with a wide range of fisheye lenses availableon the market, such as Nikon, Sigma, Omnitech-Robotics, and many others,with field of view up to 195 degrees. Contrary to the previous twotoolboxes, this toolbox may feature an automatic calibration process. Infact, both the center of distortion and the calibration points may bedetected automatically without any user intervention.

In some embodiments, before determining the external state informationof UAV based on image data captured by the monocular camera, the imagedata may be calibrated according to a pre-determined calibration model.In some embodiments, calibrating the image data may include: obtaining acoordinate mapping relationship between arbitrary point in an externalspace and a corresponding point in an image acquired by the monocularcamera unit according to the pre-established calibration model of themonocular camera; and modifying the image according to the coordinatemapping relationship. The calibration model can correspond to one ormore components or parameters of the camera unit such as characteristicsof a reflection mirror of the camera unit.

For example, in order to increase the field of view (FOV) of themonocular camera unit, the monocular camera unit may configured to takepanoramic images. For example, the monocular camera unit may furthercomprise a reflection mirror for reflecting external light rays into themonocular camera. In this case, after obtaining the current imagecaptured by the camera unit, the method of determining a state of UAVmay further include: obtaining a coordinate mapping relationship betweenarbitrary point in an external space and a corresponding point in thecurrent image captured by the monocular camera, according to apre-established calibration model corresponding to the reflectionmirror; and calibrating the current image captured by the monocularcamera unit by using the coordinate mapping.

In some instances, a curved surface of the reflection mirror may be aparabola or a hyperbola. Taking a parabola as an example, the imagingpoint of the monocular camera unit may be located on the directrix ofthe parabola, and a line connecting the focal point of the parabola andthe imaging point may be perpendicular to the directrix. Since adistance from arbitrary point on the parabola to the directrix is equalto a distance to the focal point, a calibration model may be builtaccording to this property, thereby a coordinate mapping relationshipbetween arbitrary point in an external space and a corresponding pointin the current image captured by the monocular camera unit may beobtained according to the calibration model, and the coordinate mappingrelationship may be used to calibrate when the current image of themonocular camera unit is expanded. As shown in FIG. 8, although thecurrent image 800 before calibration is severely affected by thereflection mirror, a correct image shown in FIG. 9 may be obtainedthrough column-expansion calibration.

In some instances, the image calibration process may be performed eachtime an image is captured by the monocular camera, in case a panoramictype monocular camera unit is used on the UAV. For example, beforedetermining the external state information of the UAV in step 201according to the embodiment shown in FIG. 2, a step of calibrating thecaptured image may be performed. For another example, before predictinga matching feature point in the current image in step S402 according tothe embodiment shown in FIG. 4, a step of calibrating the captured imagemay be performed to obtain the calibrated image. Alternatively, theimage calibration process may be performed after a number of images havebeen taken. Alternatively, the image calibration process may be optionaland omitted in some embodiments, for example, in case the monocularcamera unit is not a panoramic type or in case the image taken by themonocular camera unit is not distorted. In some embodiments, the imagecalibration may be performed by processors onboard the UAV.Alternatively, the image calibration may be performed by remoteterminals, for example a remote computing device. In some instances, theimage calibration may be an automatic process. For instances, the imagemay be automatically calibrated each time an image is captured by themonocular camera, or each time a predetermined number of images arecaptured by the monocular camera, or at a predetermined interval.Alternatively, the image calibration may be performed in response topredetermined events. For example, the image calibration may beperformed upon receiving a user's instruction.

The systems and methods described herein can be applied to a widevariety of movable objects. A movable object of the present disclosurecan be configured to move within any suitable environment, such as inair (e.g., a fixed-wing aircraft, a rotary-wing aircraft, or an aircrafthaving neither fixed wings nor rotary wings), in water (e.g., a ship ora submarine), on ground (e.g., a motor vehicle, such as a car, truck,bus, van, motorcycle, bicycle; a movable structure or frame such as astick, fishing pole; or a train), under the ground (e.g., a subway), inspace (e.g., a spaceplane, a satellite, or a probe), or any combinationof these environments. The movable object can be a vehicle, such as avehicle described elsewhere herein.

The UAV may be an aerial vehicle. The UAV may have one or morepropulsion units that may permit the UAV to move about in the air. Theone or more propulsion units may enable the UAV to move about one ormore, two or more, three or more, four or more, five or more, six ormore degrees of freedom. In some instances, the UAV may be able torotate about one, two, three or more axes of rotation. The axes ofrotation may be orthogonal to one another. The axes of rotation mayremain orthogonal to one another throughout the course of the UAV'sflight. The axes of rotation may include a pitch axis, roll axis, and/oryaw axis. The UAV may be able to move along one or more dimensions. Forexample, the UAV may be able to move upwards due to the lift generatedby one or more rotors. In some instances, the UAV may be capable ofmoving along a Z axis (which may be up relative to the UAV orientation),an X axis, and/or a Y axis (which may be lateral). The UAV may becapable of moving along one, two, or three axes that may be orthogonalto one another.

The UAV may be a rotorcraft. In some instances, the UAV may be amulti-rotor craft that may include a plurality of rotors. The pluralityor rotors may be capable of rotating to generate lift for the UAV. Therotors may be propulsion units that may enable the UAV to move aboutfreely through the air. The rotors may rotate at the same rate and/ormay generate the same amount of lift or thrust. The rotors mayoptionally rotate at varying rates, which may generate different amountsof lift or thrust and/or permit the UAV to rotate. In some instances,one, two, three, four, five, six, seven, eight, nine, ten, or morerotors may be provided on a UAV. The rotors may be arranged so thattheir axes of rotation being parallel to one another. In some instances,the rotors may have axes of rotation that are at any angle relative toone another, which may affect the motion of the UAV. The rotation of therotors may be driven by one or more motors coupled to the rotors.

A battery may be coupled to the UAV. The battery may be coupled to a UAVto provide power to one or more components of the UAV. The batteryprovide power to one or more propulsion units, flight controller,sensor, inertial measurement unit, communication unit, and/or any othercomponent of the UAV while coupled to the UAV. Examples of sensors ofthe UAV may include, but are not limited to, location sensors (e.g.,global positioning system (GPS) sensors, mobile device transmittersenabling location triangulation), vision sensors (e.g., imaging devicescapable of detecting visible, infrared, or ultraviolet light, such ascameras), proximity sensors (e.g., ultrasonic sensors, lidar sensors, orinfrared sensors), inertial sensors (e.g., accelerometers, gyroscopes,inertial measurement units (IMUs)), altitude sensors, pressure sensors(e.g., barometers), audio sensors (e.g., microphones), or field sensors(e.g., magnetometers, electromagnetic sensors).

A carrier may be provided to support a payload. For example, a carrierfor cameras on a UAV may be a gimbal-stabilized platform. The carriermay be generally equipped with a yaw motor, a roll motor and/or a pitchmotor such that the gimbal may rotate about one, two, or three axes ofrotation. The camera may be installed on a terminal or seat of thegimbal. By the actuation of the motors, the gimbal may independentlyadjust the yaw angle, roll angle and/or pitch angle of the camera. Thesemotors may be either driven by the battery or battery assembly installedinside of the body of UAV, or by a dedicated battery of the gimbal. Inthe actuation of these motors, noise may be generated.

The propulsion system of the UAV may include one or more rotors. A rotormay include one or more blades (e.g., one, two, three, four, or moreblades) affixed to a central shaft. The blades may be disposedsymmetrically or asymmetrically about the central shaft. The blades maybe turned by rotation of the central shaft, which can be driven by asuitable motor or engine. The blades may be configured to spin in aclockwise rotation and/or a counterclockwise rotation. The rotor may bea horizontal rotor (which may refer to a rotor having a horizontal planeof rotation), a vertically oriented rotor (which may refer to a rotorhaving a vertical plane of rotation), or a rotor tilted at anintermediate angle between the horizontal and vertical positions. Insome embodiments, horizontally oriented rotors may spin and provide liftto the movable object. Vertically oriented rotors may spin and providethrust to the movable object. Rotors oriented an intermediate anglebetween the horizontal and vertical positions may spin and provide bothlift and thrust to the movable object. One or more rotors may be used toprovide a torque counteracting a torque produced by the spinning ofanother rotor.

The movable object may be capable of moving freely within theenvironment with respect to six degrees of freedom (e.g., three degreesof freedom in translation and three degrees of freedom in rotation).Alternatively, the movement of the movable object can be constrainedwith respect to one or more degrees of freedom, such as by apredetermined path, track, or orientation. The movement can be actuatedby any suitable actuation mechanism, such as an engine or a motor. Theactuation mechanism of the movable object can be powered by any suitableenergy source, such as electrical energy, magnetic energy, solar energy,wind energy, gravitational energy, chemical energy, nuclear energy, orany suitable combination thereof. The movable object may beself-propelled via a propulsion system, as described elsewhere herein.The propulsion system may optionally run on an energy source, such aselectrical energy, magnetic energy, solar energy, wind energy,gravitational energy, chemical energy, nuclear energy, or any suitablecombination thereof.

In some instances, the movable object can be an aerial vehicle. Forexample, aerial vehicles may be fixed-wing aircraft (e.g., airplane,gliders), rotary-wing aircraft (e.g., helicopters, rotorcraft), aircrafthaving both fixed wings and rotary wings, or aircraft having neither(e.g., blimps, hot air balloons). An aerial vehicle can beself-propelled, such as self-propelled through the air. A self-propelledaerial vehicle can utilize a propulsion system, such as a propulsionsystem including one or more engines, motors, wheels, axles, magnets,rotors, propellers, blades, nozzles, or any suitable combinationthereof. In some instances, the propulsion system can be used to enablethe movable object to take off from a surface, land on a surface,maintain its current position and/or orientation (e.g., hover), changeorientation, and/or change position.

The movable object can be controlled remotely by a user or controlledlocally by an occupant within or on the movable object. The movableobject may be controlled remotely via an occupant within a separatevehicle. In some embodiments, the movable object is an unmanned movableobject, such as a UAV. An unmanned movable object, such as a UAV, maynot have an occupant onboard the movable object. The movable object canbe controlled by a human or an autonomous control system (e.g., acomputer control system), or any suitable combination thereof. Themovable object can be an autonomous or semi-autonomous robot, such as arobot configured with an artificial intelligence.

The movable object can have any suitable size and/or dimensions. In someembodiments, the movable object may be of a size and/or dimensions tohave a human occupant within or on the vehicle. Alternatively, themovable object may be of size and/or dimensions smaller than thatcapable of having a human occupant within or on the vehicle. The movableobject may be of a size and/or dimensions suitable for being lifted orcarried by a human. Alternatively, the movable object may be larger thana size and/or dimensions suitable for being lifted or carried by ahuman. In some instances, the movable object may have a maximumdimension (e.g., length, width, height, diameter, diagonal) of less thanor equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. Themaximum dimension may be greater than or equal to about: 2 cm, 5 cm, 10cm, 50 cm, 1 m, 2 m, 5 m, or 10 m. For example, the distance betweenshafts of opposite rotors of the movable object may be less than orequal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m, or 10 m.Alternatively, the distance between shafts of opposite rotors may begreater than or equal to about: 2 cm, 5 cm, 10 cm, 50 cm, 1 m, 2 m, 5 m,or 10 m.

In some embodiments, the movable object may have a volume of less than100 cm×100 cm×100 cm, less than 50 cm×50 cm×30 cm, or less than 5 cm×5cm×3 cm. The total volume of the movable object may be less than orequal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40 cm³, 50cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³, 300 cm³,500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³3, 1 m³, or10 m³. Conversely, the total volume of the movable object may be greaterthan or equal to about: 1 cm³, 2 cm³, 5 cm³, 10 cm³, 20 cm³, 30 cm³, 40cm³, 50 cm³, 60 cm³, 70 cm³, 80 cm³, 90 cm³, 100 cm³, 150 cm³, 200 cm³,300 cm³, 500 cm³, 750 cm³, 1000 cm³, 5000 cm³, 10,000 cm³, 100,000 cm³,1 m³, or 10 m³.

In some embodiments, the movable object may have a footprint (which mayrefer to the lateral cross-sectional area encompassed by the movableobject) less than or equal to about: 32,000 cm², 20,000 cm², 10,000 cm²,1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm². Conversely, thefootprint may be greater than or equal to about: 32,000 cm², 20,000 cm²,10,000 cm², 1,000 cm², 500 cm², 100 cm², 50 cm², 10 cm², or 5 cm².

In some instances, the movable object may weigh no more than 1000 kg.The weight of the movable object may be less than or equal to about:1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60 kg, 50kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10 kg, 9 kg,8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1 kg, 0.05 kg,or 0.01 kg. Conversely, the weight may be greater than or equal toabout: 1000 kg, 750 kg, 500 kg, 200 kg, 150 kg, 100 kg, 80 kg, 70 kg, 60kg, 50 kg, 45 kg, 40 kg, 35 kg, 30 kg, 25 kg, 20 kg, 15 kg, 12 kg, 10kg, 9 kg, 8 kg, 7 kg, 6 kg, 5 kg, 4 kg, 3 kg, 2 kg, 1 kg, 0.5 kg, 0.1kg, 0.05 kg, or 0.01 kg.

In some embodiments, a movable object may be small relative to a loadcarried by the movable object. The load may include a payload and/or acarrier, as described in further detail elsewhere herein. In someexamples, a ratio of a movable object weight to a load weight may begreater than, less than, or equal to about 1:1. In some instances, aratio of a movable object weight to a load weight may be greater than,less than, or equal to about 1:1. Optionally, a ratio of a carrierweight to a load weight may be greater than, less than, or equal toabout 1:1. When desired, the ratio of an movable object weight to a loadweight may be less than or equal to: 1:2, 1:3, 1:4, 1:5, 1:10, or evenless. Conversely, the ratio of a movable object weight to a load weightcan also be greater than or equal to: 2:1, 3:1, 4:1, 5:1, 10:1, or evengreater.

In some embodiments, the movable object may have low energy consumption.For example, the movable object may use less than about: 5 W/h, 4 W/h, 3W/h, 2 W/h, 1 W/h, or less. In some instances, a carrier of the movableobject may have low energy consumption. For example, the carrier may useless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less. Optionally,a payload of the movable object may have low energy consumption, such asless than about: 5 W/h, 4 W/h, 3 W/h, 2 W/h, 1 W/h, or less.

FIG. 10 illustrates an unmanned aerial vehicle (UAV) 1000, in accordancewith embodiments of the present disclosure. The UAV may be an example ofa movable object as described herein. The UAV 1000 can include apropulsion system having four rotors 1002, 1004, 1006, and 1008. Anynumber of rotors may be provided (e.g., one, two, three, four, five,six, or more). The rotors, rotor assemblies, or other propulsion systemsof the unmanned aerial vehicle may enable the unmanned aerial vehicle tohover/maintain position, change orientation, and/or change location. Thedistance between shafts of opposite rotors can be any suitable length1010. For example, the length 1010 can be less than or equal to 2 m, orless than equal to 5 m. In some embodiments, the length 1010 can bewithin a range from 40 cm to 1 m, from 10 cm to 2 m, or from 5 cm to 5m. Any description herein of a UAV may apply to a movable object, suchas a movable object of a different type, and vice versa. The UAV may usean assisted takeoff system or method as described herein.

In some embodiments, the movable object can be configured to carry aload. The load can include one or more of passengers, cargo, equipment,instruments, and the like. The load can be provided within a housing.The housing may be separate from a housing of the movable object, or bepart of a housing for a movable object. Alternatively, the load can beprovided with a housing while the movable object does not have ahousing. Alternatively, portions of the load or the entire load can beprovided without a housing. The load can be rigidly fixed relative tothe movable object. Optionally, the load can be movable relative to themovable object (e.g., translatable or rotatable relative to the movableobject). The load can include a payload and/or a carrier, as describedelsewhere herein.

In some embodiments, the movement of the movable object, carrier, andpayload relative to a fixed reference frame (e.g., the surroundingenvironment) and/or to each other, can be controlled by a terminal. Theterminal can be a remote control device at a location distant from themovable object, carrier, and/or payload. The terminal can be disposed onor affixed to a support platform. Alternatively, the terminal can be ahandheld or wearable device. For example, the terminal can include asmartphone, tablet, laptop, computer, glasses, gloves, helmet,microphone, or suitable combinations thereof. The terminal can include auser interface, such as a keyboard, mouse, joystick, touchscreen, ordisplay. Any suitable user input can be used to interact with theterminal, such as manually entered commands, voice control, gesturecontrol, or position control (e.g., via a movement, location or tilt ofthe terminal).

The terminal can be used to control any suitable state of the movableobject, carrier, and/or payload. For example, the terminal can be usedto control the position and/or orientation of the movable object,carrier, and/or payload relative to a fixed reference from and/or toeach other. In some embodiments, the terminal can be used to controlindividual elements of the movable object, carrier, and/or payload, suchas the actuation assembly of the carrier, a sensor of the payload, or anemitter of the payload. The terminal can include a wirelesscommunication device adapted to communicate with one or more of themovable object, carrier, or payload.

The terminal can include a suitable display unit for viewing informationof the movable object, carrier, and/or payload. For example, theterminal can be configured to display information of the movable object,carrier, and/or payload with respect to position, translationalvelocity, translational acceleration, orientation, angular velocity,angular acceleration, or any suitable combinations thereof. In someembodiments, the terminal can display information provided by thepayload, such as data provided by a functional payload (e.g., imagesrecorded by a camera or other image capturing device).

Optionally, the same terminal may both control the movable object,carrier, and/or payload, or a state of the movable object, carrierand/or payload, as well as receive and/or display information from themovable object, carrier and/or payload. For example, a terminal maycontrol the positioning of the payload relative to an environment, whiledisplaying image data captured by the payload, or information about theposition of the payload. Alternatively, different terminals may be usedfor different functions. For example, a first terminal may controlmovement or a state of the movable object, carrier, and/or payload whilea second terminal may receive and/or display information from themovable object, carrier, and/or payload. For example, a first terminalmay be used to control the positioning of the payload relative to anenvironment while a second terminal displays image data captured by thepayload. Various communication modes may be utilized between a movableobject and an integrated terminal that both controls the movable objectand receives data, or between the movable object and multiple terminalsthat both control the movable object and receives data. For example, atleast two different communication modes may be formed between themovable object and the terminal that both controls the movable objectand receives data from the movable object.

FIG. 11 illustrates a movable object 1100 including a carrier 1102 and apayload 1104, in accordance with embodiments of the present disclosure.Although the movable object 1100 is depicted as an aircraft, thisdepiction is not intended to be limiting, and any suitable type ofmovable object can be used, as previously described herein. One of skillin the art would appreciate that any of the embodiments described hereinin the context of aircraft systems can be applied to any suitablemovable object (e.g., an UAV). In some instances, the payload 1104 maybe provided on the movable object 1100 without requiring the carrier1102. The movable object 1100 may include propulsion mechanisms 1106, asensing system 1108, and a communication system 1110.

The propulsion mechanisms 1106 can include one or more of rotors,propellers, blades, engines, motors, wheels, axles, magnets, or nozzles,as previously described. The movable object may have one or more, two ormore, three or more, or four or more propulsion mechanisms. Thepropulsion mechanisms may all be of the same type. Alternatively, one ormore propulsion mechanisms can be different types of propulsionmechanisms. The propulsion mechanisms 1106 can be mounted on the movableobject 1100 using any suitable means, such as a support element (e.g., adrive shaft) as described elsewhere herein. The propulsion mechanisms1106 can be mounted on any suitable portion of the movable object 1100,such on the top, bottom, front, back, sides, or suitable combinationsthereof.

In some embodiments, the propulsion mechanisms 1106 can enable themovable object 1100 to take off vertically from a surface or landvertically on a surface without requiring any horizontal movement of themovable object 1100 (e.g., without traveling down a runway). Optionally,the propulsion mechanisms 1106 can be operable to permit the movableobject 1100 to hover in the air at a specified position and/ororientation. One or more of the propulsion mechanisms 1100 may becontrolled independently of the other propulsion mechanisms.Alternatively, the propulsion mechanisms 1100 can be configured to becontrolled simultaneously. For example, the movable object 1100 can havemultiple horizontally oriented rotors that can provide lift and/orthrust to the movable object. The multiple horizontally oriented rotorscan be actuated to provide vertical takeoff, vertical landing, andhovering capabilities to the movable object 1100. In some embodiments,one or more of the horizontally oriented rotors may spin in a clockwisedirection, while one or more of the horizontally rotors may spin in acounterclockwise direction. For example, the number of clockwise rotorsmay be equal to the number of counterclockwise rotors. The rotation rateof each of the horizontally oriented rotors can be varied independentlyin order to control the lift and/or thrust produced by each rotor, andthereby adjust the spatial disposition, velocity, and/or acceleration ofthe movable object 1100 (e.g., with respect to up to three degrees oftranslation and up to three degrees of rotation).

The sensing system 1108 can include one or more sensors that may sensethe spatial disposition, velocity, and/or acceleration of the movableobject 1100 (e.g., with respect to up to three degrees of translationand up to three degrees of rotation). The one or more sensors caninclude global positioning system (GPS) sensors, motion sensors,inertial sensors, proximity sensors, or image sensors. The sensing dataprovided by the sensing system 1108 can be used to control the spatialdisposition, velocity, and/or orientation of the movable object 1100(e.g., using a suitable processing unit and/or control module, asdescribed below). Alternatively, the sensing system 1108 can be used toprovide data regarding the environment surrounding the movable object,such as weather conditions, proximity to potential obstacles, locationof geographical features, location of manmade structures, and the like.

The communication system 1110 enables communication with terminal 1112having a communication system 1114 via wireless signals 1116. Thecommunication systems 1110, 1114 may include any number of transmitters,receivers, and/or transceivers suitable for wireless communication. Thecommunication may be one-way communication, such that data can betransmitted in only one direction. For example, one-way communicationmay involve only the movable object 1100 transmitting data to theterminal 1112, or vice-versa. The data may be transmitted from one ormore transmitters of the communication system 1110 to one or morereceivers of the communication system 1112, or vice-versa.Alternatively, the communication may be two-way communication, such thatdata can be transmitted in both directions between the movable object1100 and the terminal 1112. The two-way communication can involvetransmitting data from one or more transmitters of the communicationsystem 1110 to one or more receivers of the communication system 1114,and vice-versa.

In some embodiments, the terminal 1112 can provide control data to oneor more of the movable object 1100, carrier 1102, and payload 1104 andreceive information from one or more of the movable object 1100, carrier1102, and payload 1104 (e.g., position and/or motion information of themovable object, carrier or payload; data sensed by the payload such asimage data captured by a payload camera). In some instances, controldata from the terminal may include instructions for relative positions,movements, actuations, or controls of the movable object, carrier and/orpayload. For example, the control data may result in a modification ofthe location and/or orientation of the movable object (e.g., via controlof the propulsion mechanisms 1106), or a movement of the payload withrespect to the movable object (e.g., via control of the carrier 1102).The control data from the terminal may result in control of the payload,such as control of the operation of a camera or other image capturingdevice (e.g., taking still or moving pictures, zooming in or out,turning on or off, switching imaging modes, change image resolution,changing focus, changing depth of field, changing exposure time,changing viewing angle or field of view). In some instances, thecommunications from the movable object, carrier and/or payload mayinclude information from one or more sensors (e.g., of the sensingsystem 1108 or of the payload 1104). The communications may includesensed information from one or more different types of sensors (e.g.,GPS sensors, motion sensors, inertial sensor, proximity sensors, orimage sensors). Such information may pertain to the position (e.g.,location, orientation), movement, or acceleration of the movable object,carrier and/or payload. Such information from a payload may include datacaptured by the payload or a sensed state of the payload. The controldata provided transmitted by the terminal 1112 can be configured tocontrol a state of one or more of the movable object 1100, carrier 1102,or payload 1104. Alternatively or in combination, the carrier 1102 andpayload 1104 can also each include a communication module configured tocommunicate with terminal 1112, such that the terminal can communicatewith and control each of the movable object 1100, carrier 1102, andpayload 1104 independently.

In some embodiments, the movable object 1100 can be configured tocommunicate with another remote device in addition to the terminal 1112,or instead of the terminal 1112. The terminal 1112 may also beconfigured to communicate with another remote device as well as themovable object 1100. For example, the movable object 1100 and/orterminal 1112 may communicate with another movable object, or a carrieror payload of another movable object. When desired, the remote devicemay be a second terminal or other computing device (e.g., computer,laptop, tablet, smartphone, or other mobile device). The remote devicecan be configured to transmit data to the movable object 1100, receivedata from the movable object 1100, transmit data to the terminal 1112,and/or receive data from the terminal 1112. Optionally, the remotedevice can be connected to the Internet or other telecommunicationsnetwork, such that data received from the movable object 1100 and/orterminal 1112 can be uploaded to a website or server.

FIG. 12 is a schematic illustration by way of block diagram of a system1200 for controlling a movable object, in accordance with embodiments ofthe present disclosure. The system 1200 can be used in combination withany suitable embodiment of the systems, devices, and methods disclosedherein. The system 1200 can include a sensing module 1202, processingunit 1204, non-transitory computer readable medium 1206, control module1208, and communication module 1210.

The sensing module 1202 can utilize different types of sensors thatcollect information relating to the movable objects in different ways.Different types of sensors may sense different types of signals orsignals from different sources. For example, the sensors can includeinertial sensors, GPS sensors, proximity sensors (e.g., lidar), orvision/image sensors (e.g., a camera). The sensing module 1202 can beoperatively coupled to a processing unit 1204 having a plurality ofprocessors. In some embodiments, the sensing module can be operativelycoupled to a transmission module 1212 (e.g., a Wi-Fi image transmissionmodule) configured to directly transmit sensing data to a suitableexternal device or system. For example, the transmission module 1212 canbe used to transmit images captured by a camera of the sensing module1202 to a remote terminal.

The processing unit 1204 can have one or more processors, such as aprogrammable processor (e.g., a central processing unit (CPU)). Theprocessing unit 1204 can be operatively coupled to a non-transitorycomputer readable medium 1206. The non-transitory computer readablemedium 1206 can store logic, code, and/or program instructionsexecutable by the processing unit 1204 for performing one or more steps.The non-transitory computer readable medium can include one or morememory units (e.g., removable media or external storage such as an SDcard or random access memory (RAM)). In some embodiments, data from thesensing module 1202 can be directly conveyed to and stored within thememory units of the non-transitory computer readable medium 1206. Thememory units of the non-transitory computer readable medium 1206 canstore logic, code and/or program instructions executable by theprocessing unit 1204 to perform any suitable embodiment of the methodsdescribed herein. For example, the processing unit 1204 can beconfigured to execute instructions causing one or more processors of theprocessing unit 1204 to analyze sensing data produced by the sensingmodule. The memory units can store sensing data from the sensing moduleto be processed by the processing unit 1204. In some embodiments, thememory units of the non-transitory computer readable medium 1206 can beused to store the processing results produced by the processing unit1204.

In some embodiments, the processing unit 1204 can be operatively coupledto a control module 1208 configured to control a state of the movableobject. For example, the control module 1208 can be configured tocontrol the propulsion mechanisms of the movable object to adjust thespatial disposition, velocity, and/or acceleration of the movable objectwith respect to six degrees of freedom. Alternatively or in combination,the control module 1208 can control one or more of a state of a carrier,payload, or sensing module.

The processing unit 1204 can be operatively coupled to a communicationmodule 1210 configured to transmit and/or receive data from one or moreexternal devices (e.g., a terminal, display device, or other remotecontroller). Any suitable means of communication can be used, such aswired communication or wireless communication. For example, thecommunication module 1210 can utilize one or more of local area networks(LAN), wide area networks (WAN), infrared, radio, WiFi, point-to-point(P2P) networks, telecommunication networks, cloud communication, and thelike. Optionally, relay stations, such as towers, satellites, or mobilestations, can be used. Wireless communications can be proximitydependent or proximity independent. In some embodiments, line-of-sightmay or may not be required for communications. The communication module1210 can transmit and/or receive one or more of sensing data from thesensing module 1202, processing results produced by the processing unit1204, predetermined control data, user commands from a terminal orremote controller, and the like.

The components of the system 1200 can be arranged in any suitableconfiguration. For example, one or more of the components of the system1200 can be located on the movable object, carrier, payload, terminal,sensing system, or an additional external device in communication withone or more of the above. Additionally, although FIG. 12 depicts asingle processing unit 1204 and a single non-transitory computerreadable medium 1206, one of skill in the art would appreciate that thisis not intended to be limiting, and that the system 1200 can include aplurality of processing units and/or non-transitory computer readablemedia. In some embodiments, one or more of the plurality of processingunits and/or non-transitory computer readable media can be situated atdifferent locations, such as on the movable object, carrier, payload,terminal, sensing module, additional external device in communicationwith one or more of the above, or suitable combinations thereof, suchthat any suitable aspect of the processing and/or memory functionsperformed by the system 1200 can occur at one or more of theaforementioned locations.

While some embodiments of the present disclosure have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the disclosure. It should beunderstood that various alternatives to the embodiments of thedisclosure described herein may be employed in practicing thedisclosure. It is intended that the following claims define the scope ofthe disclosure and that methods and structures within the scope of theseclaims and their equivalents be covered thereby.

What is claimed is:
 1. A method for determining an external state of anunmanned aerial vehicle (UAV) comprising: obtaining historical externalstate information of the UAV; obtaining a current image and a historicalimage captured before the current image; predicting, according thehistorical external state information, a matching feature point in thecurrent image that corresponds to a target feature point in thehistorical image; and determining the external state of the UAV based onthe matching feature point.
 2. The method of claim 1, wherein theexternal state includes a distance of the UAV relative to an externalobject.
 3. The method of claim 1, wherein obtaining the historicalexternal state information includes obtaining at least one of distanceinformation of the UAV relative to one or more external objects,position information of the one or more external object, positioninformation of the UAV, or a feature point associated with a set ofcoordinates under a reference model.
 4. The method of claim 1, whereinobtaining the current image and the historical image includes obtainingthe current image and the historical image from a series of imagescaptured by a monocular camera.
 5. The method of claim 1, whereinobtaining the current image includes obtaining the current image in realtime from a monocular camera.
 6. The method of claim 1, whereinobtaining the historical image includes reading the historical imagefrom a memory onboard the UAV that stores the series of captured images.7. The method of claim 1, wherein obtaining the historical imageincludes receiving the historical image from a remote terminal.
 8. Themethod of claim 1, wherein: obtaining the historical external stateinformation includes obtaining information characterizing a motionvector associated with the UAV; and predicting the matching featurepoint includes predicting the matching feature point according to themotion vector.
 9. The method of claim 8, wherein obtaining theinformation characterizing the motion vector associated with the UAVincludes obtaining at least one of a direction or a speed of the UAV.10. The method of claim 1, wherein predicting the matching feature pointincludes: predicting a coordinate position of the target feature pointin the current image according to the historical external stateinformation; and selecting, from one or more feature points in thecurrent image, a feature point that is closest to the predictedcoordinate position as the matching feature point.
 11. A system forcontrolling an unmanned aerial vehicle comprising: a processor; and amemory storing program instructions that, when executed by theprocessor, cause the processor to: obtain historical external stateinformation of the UAV; obtain a current image and a historical imagecaptured before the current image; predict, according the historicalexternal state information, a matching feature point in the currentimage that corresponds to a target feature point in the historicalimage; and determine the external state of the UAV based on the matchingfeature point.
 12. The system of claim 11, wherein the external stateincludes a distance of the UAV relative to an external object.
 13. Thesystem of claim 11, wherein the historical external state informationincludes at least one of distance information of the UAV relative to oneor more external objects, position information of the one or moreexternal object, position information of the UAV, or a feature pointassociated with a set of coordinates under a reference model.
 14. Thesystem of claim 11, wherein the instructions further cause the processorto obtain the current image and the historical image from a series ofimages captured by a monocular camera.
 15. The system of claim 11,wherein the instructions further cause the processor to obtain thecurrent image in real time from a monocular camera.
 16. The system ofclaim 11, wherein the instructions further cause the processor to readthe historical image from a storage memory onboard the UAV that storesthe series of captured images.
 17. The system of claim 11, wherein theinstructions further cause the processor to receive the historical imagefrom a remote terminal.
 18. The system of claim 11, wherein theinstructions further cause the processor to: obtain informationcharacterizing a motion vector associated with the UAV as the historicalexternal state information; and predict the matching feature pointaccording to the motion vector.
 19. The system of claim 18, wherein themotion vector associated with the UAV includes at least one of adirection or a speed of the UAV.
 20. The system of claim 11, wherein theinstructions further cause the processor to: predict a coordinateposition of the target feature point in the current image according tothe historical external state information; and select, from one or morefeature points in the current image, a feature point that is closest tothe predicted coordinate position as the matching feature point.